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INTRODUCTION 


Many  traditional  structural  materials,  which  are  homogeneous  and  isotropic,  differ  from 
composite  materials  which  have  extensive  intrinsic  statistical  variability  in  many  material  proper¬ 
ties.  This  variability,  particularly  important  to  strength  properties,  is  due  not  only  to 
inhomogeneity  and  anisotropy,  but  also  to  the  basic  brittleness  of  many  matrices  and  most 
fibers  and  to  the  potential  for  property  mismatch  between  the  components.  Because  of  this 
inherent  statistical  variability,  careful  statistical  analysis  of  composite  material  properties  is  not 
only  more  important  but  is  also  more  complex  than  for  traditional  structures. 


This  report  addresses  this  issue  by  discussing  the  methodologies  and  their  sequence  of 
application  for  obtaining  statistical  material  property  values  (basis  values).  A  more  detailed 
analysis  showing  the  various  operations  required  for  computation  of  the  basis  value  is  pre¬ 
sented  by  the  authors  in  the  statistics  chapter  of  the  MIL-17  Handbook.1  The  procedures  in 
this  handbook  required  substantial  research  efforts  in  order  to  accommodate  various  require¬ 
ments  (e.g.,  small  samples,  batch-to-batch  variability,  and  tolerance  limits)  for  obtaining  the 
basis  values.  Guidance  in  selection  of  the  methodology  came  from  the  needs  of  the  military, 
aircraft  industry,  and  the  Federal  Aviation  Administration  (FAA).  Some  of  the  procedures 
include  determination  of  outliers,  selection  of  statistical  models,  tests  for  batch-to-batch  varia¬ 
tion,  single  and  multi-batch  models  for  basis  value  computation,  and  nonparametric  methods. 
In  Figure  1,  a  flowchart  is  shown  outlining  the  sequence  of  operations. 


Figure  1 .  Flowchart  illustrating  computational  procedures  for  statistically-based  material  properties. 


1.  MIL-HDBK-17B.  Polymer  Matrix  Composites.  Naval  Publications  and  Forms  Center,  5801  Tabor  Avenue,  Philadelphia,  Pennsylvania 
19120,  v.  1,  February  1988. 
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An  important  application  of  the  basis  property  value  is  to  the  design  of  composite  aircraft 
structures  where  a  design  allowable  is  developed  from  this  value.  The  process  usually 
involves  a  reduction  in  the  basis  values  in  order  to  represent  a  specific  application  of  the  com¬ 
posite  material  in  a  structure  (for  example,  a  structure  with  a  bolt  hole  for  a  particular  test 
and  environmental  condition).  One  common  approach  in  the  design  process  requires  the 
design  allowable  be  divided  by  the  maximum  applied  stress  or  strain  and  the  result  to  be 
greater  than  one.  The  basis  value  is  also  used  in  qualifying  new  composite  material  systems 
to  be  used  in  the  manufacture  of  aircraft.  In  this  case,  the  values  are  obtained  from  an 
extensive  test  matrix,  including  both  loading  and  environmental  conditions.  The  value  also 
provides  guidance  in  selecting  material  systems  for  specific  design  requirements. 

This  report  also  shows  how  material  strength  variability  and  the  number  of  test  specimens 
can  affect  the  determination  of  reliability  numbers.  Methods  are  presented  for  obtaining  pro¬ 
tection  against  this  situation  by  providing  a  tolerance  limit  value  on  a  stress  corresponding  to 
a  high  reliability.  A  comparison  between  deterministic  and  statistical  reliability  estimates  dem¬ 
onstrates  the  inadequacy  of  the  deterministic  approach.  A  case  study  is  presented  describing 
the  recommended  procedures  outlined  in  the  MIL-17  Handbook  for  determining  statistically- 
based  material  property  values. 


RELIABILITY  ESTIMATES 


Sample  Size  -  Variability 

The  importance  of  determining  a  tolerance  limit  on  a  percentile  value  is  graphically  dis¬ 
played  in  Figures  2a  and  2b.  The  cumulative  distribution  function  (CDF)  of  the  standard 
normal  (mean  equals  0,  standard  deviation  1)  is  plotted  for  sample  sizes  of  10  and  50,  using 
25  randomly  selected  sets  of  data.  In  Figure  2a,  for  n  equals  10,  the  spread  in  the  percen¬ 
tile  is  2.1  for  the  10th  percentile.  In  Figure  2b,  for  n  equals  50,  the  spread  is  0.7  for  the 
same  percentile.  The  results  show  the  relative  uncertainty  associated  with  small  sample  sizes 
when  computing  reliability  values.  The  range  in  the  percentile  can  also  depend  on  the 
amount  of  variability  in  the  data  (i.e.,  the  variance). 

Often  in  structural  design,  a  design  allowable  value  is  obtained  from  the  basis  value.  A 
design  allowable  is  an  experimentally  determined  acceptable  stress  value  for  a  material  (called 
an  allowable  stress).  The  allowable  is  a  function  of  the  material  basis  value,  layup,  damage 
tolerance,  open  holes,  and  other  factors.  It  is  usually  numerically  determined  for  some 
critical  stress  region  located  within  the  structure.  In  using  the  allowable,  it  is  required  that 
the  critical  stress  be  less  than  a  proportion  (margin  of  safety)  of  the  allowable  stress  value. 
Determining  a  property  value  from  only  10  strength  tests  using  90%  reliability  estimates  with¬ 
out  confidence  in  the  assertion  could  result  in  a  nonconservative  design  situation.  In  order 
to  prevent  this  occurrence  and  provide  a  guarantee  of  the  reliability  value,  a  tolerance  limit 
(i.e.,  a  lower  confidence  bound)  on  the  percentile  is  recommended.  The  MIL- 17  Handbook 
statistics  chapter  describes  methods  for  obtaining  basis  values  for  a  prescribed  tolerance  limit. 

Definition  of  the  B-Basis  Value 

The  B-Basis  value  is  a  random  variable  where  an  observed  basis  value  from  a  sample 
(data  set)  will  be  less  than  the  10th  percentile  of  the  population  with  a  probability  of  0.95. 

In  Figures  3a  and  3b,  a  graphical  display  is  shown  of  the  basis  value  probability  density  func¬ 
tions  for  random  samples  of  n  equals  10  and  50,  respectively.  Samples  are  from  the  same 
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population  as  in  Figures  2a  and  2b.  The  vertical  dotted  lines  represent  the  location  of  the 
population  10th  percentile  (Xo.io)-  The  probability  density  functiou  of  the  population  is  also 
displayed  in  the  figures.  Note  that  95%  of  the  time  the  basis  value  is  less  than  X0.io-  The 
graphical  display  of  the  basis  value  density  function  shows  much  less  dispersion  for  n  equals 
50  than  for  n  equals  10;  therefore,  small  sample  sizes  often  result  in  very  conservative  esti¬ 
mates  of  the  basis  value. 


Figure  2.  Semple  size  effect  on  reliability.  Random 
data  sets  of  size  n  from  a  normal  distribution. 
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Figure  3.  Basis  value  probability  density  function. 


STATISTICAL  METHODS  -  MATERIAL  PROPERTY  VALUES 


Flowchart  Guidelines 

Since  the  statistical  procedures  and  the  flowchart  (see  Figure  1)  have  been  published  in 
the  MIL- 17  Handbook,1,2  this  report  will  only  present  a  brief  description  of  the  methods. 

The  purpose,  interpretation  of  results,  and  the  order  of  application  suggested  by  the  flowchart 
will  be  the  primary  objective  of  this  report.  The  authors  have  written  a  computer  code 
which  performs  the  necessary  computations  for  obtaining  the  basis  values  as  described  in  the 
flowchart.  The  code  is  available  on  a  diskette,  which  can  be  used  or.  various  computers, 
including  PCs  that  are  IBM  compatible.  Both  the  executable  and  source  code  are  on  the  disk¬ 
ette.  This  code  is  available  free  of  charge  from  the  authors.  The  flowchart  capability  was 
tested  by  applying  the  recommended  procedures  using  both  real  and  simulated  data  sets.  The 
results  of  the  simulations  showed  at  least  95%  of  computed  values  were  less  than  the  known 
10%  point,  this  is  consistent  with  the  definitions  of  “B”-basis  value,  see  References  1  and  2. 

2.  NEAL,  D.  Mj  V  ANGEL,  M.  G.,  and  TODT,  F.  Determination  of  Statistically  Based  Composite  Material  Properties  in  Engineered  Materials 
Handbook.  Composites,  C.  A.  Dostal,  ed.,  American  Society  of  Metals  Press,  Metals  Park,  Ohio,  v.  1,  1987. 
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The  flowchart  has  two  directions  of  operations;  one  is  for  the  single  batch  (sample)  and 
the  other  is  for  the  multi-batch  case.  A  batch  could  represent  specimens  made  from  a  manu¬ 
factured  sheet  of  composite  material  representing  a  roll  of  prepreg  material.  Published 
MIL- 17  Handbook  basis  values  are  usually  obtained  from  five  batches  of  six  specimens  each. 

Initially,  let  us  assume  the  user  of  the  flowchart  has  only  a  single  batch,  or  more  than 
one  batch  but  that  the  batches  can  be  pooled  so  that  a  single  sample  analysis  can  be  applied. 
The  first  operation  (see  Figure  1)  is  to  determine  if  outliers  exist  in  the  data  set.  A  more 
detailed  discussion  of  outlier  detection  schemes  and  applications  are  published  in  Reference  3. 
The  method  selected  is  called  the  Maximum  Normed  Residual  (MNR)  procedure3 4  and  is  pub¬ 
lished  in  the  MIL-17  Handbook.  It  is  simple  to  apply  and  performs  reasonably  well,  even 
though  it  assumes  that  the  data  is  from  a  symmetric  distribution.  The  analysis  requires 
obtaining  an  ordered  array  of  normed  residuals,  written  as: 

NR;  =  (Xj  -  x)/s,  i  =  1,  — n,  (1) 

where  x  is  the  mean,  s  is  the  standard  deviation  (SD),  and  n  is  the  sample  size.  If  the 
maximum  absolute  value  of  NRj  (MNR)  is  less  than  some  critical  value  (CV),1,2  then  no 
outliers  exist.  If  MNR  is  greater  than  CV,  then  an  outlier  X  is  determined  from  the  largest 
NR;  value. 

Outlying  test  results  are  substantially  different  from  the  primary  data.  For  example, 
assume  that  the  data  set  contains  16  strength  values  and  15  range  from  150  to  200  ksi,  while 
the  other  is  80  ksi.  The  MNR  method  would  identify  the  80  ksi  value  to  be  an  outlier. 

The  80  ksi  specimen  should  be  examined  for  problems  in  fabrication  and  testing.  If  a  ratio¬ 
nale  is  determined  for  rejecting  this  test  result,  then  do  not  include  the  outlying  test  value  in 
the  data  set  when  obtaining  the  basis  value.  If  there  is  no  rationale  for  rejection,  the  outlier 
should  remain  unless  the  test  engineer  believes  that  a  nondetectable  error  exists. 

It  is  important  to  identify  the  existence  of  outliers,  but  it  is  also  of  equal  importance  to 
resist  removing  the  values  unless  a  rationale  has  been  established.  Leaving  in,  or  arbitrary 
removal  of,  outlying  values  can  adversely  affect  the  statistical  model  selection  process  and,  con¬ 
sequently,  the  basis  value  computation.  An  outlier  in  a  data  set  will  usually  result  in  a  larger 
variance  and  a  possible  shift  in  the  mean  when  compared  with  the  same  data  without  the  out¬ 
lier.  The  amount  of  shift  and  the  variance  increase  depends  on  the  severity  of  the  outlier 
(distance  removed  from  the  primary  data  set).  It  is  suggested  that  for  small  samples  (n  is 
less  than  20)  critical  values  corresponding  to  a  10%  significance  level  be  used1,2  in  order  to 
identify  outlying  values.  If  the  sample  is  greater  than  20,  then  use  the  5%  level.  It  is  often 
difficult  to  test  for  outliers  when  there  is  a  limited  amount  of  data;  therefore,  the  10%  level 
will  provide  additional  power  to  detect  outliers.  This  level  will  also  result  in  more  chance  of 
incorrectly  identifying  outliers.  Outliers  can  be  incorrectly  identified  from  data  sets  with 
highly  skewed  distributions;  therefore,  it  is  suggested  the  box-plot  method1,3  be  applied  for 
determining  outliers  in  this  situation. 


3.  NEAL,  D.  M.,  and  SPIRIDIGLIOZZI,  L.  An  Efficient  Method  for  Determining  “A”  aid  “B”  Allowables.  ARO  83-2  in  Proceedings  of  the 
28th  Conference  on  the  Design  of  Experiments  in  Army  Research,  Army  Research  Office,  1983,  p.  199-235. 

4.  STEFANSKY,  W.  Rejecting  Outtien  in  Factorial  Designs.  Technometrics,  v.  14,  1972,  p.  469479. 
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Goodnes*-of-Fit  Test  -  Distribution  Function 

Referring  to  Figure  1,  the  next  step  is  to  identify  an  acceptable  model  for  representing 
the  data.  In  the  order  of  preference,  the  three  candidate  models  are  Weibull,  normal,  and 
the  nonparametric  method.  The  Weibull  model  is: 

Fw(x)  =  1  -  exp[(x/a/],  (2) 

where  x  is  greater  than  0,  a  is  the  scale  parameter,  and  /?  is  the  shape  parameter.  This 
model  is  considered  first  in  the  ordering  of  the  test  procedures.  The  Anderson-Darling  (AD) 
goodness-of-fit  test  statistic1,5  is  suggested  for  identifying  the  model  because  it  emphasizes  dis¬ 
crepancies  in  the  tail  regions  between  the  cumulative  distribution  function  of  the  data  and  the 
cumulative  distribution  function  of  the  model.  This  is  more  desirable  than  evaluating  the  dis¬ 
tributional  assumptions  near  the  mean,  since  reliability  estimates  are  usually  measured  in  the 
tail  regions.  The  Anderson- Darling  test  statistic  and  the  observed  significance  levels  computa¬ 
tions  are  described  in  References  1  and  2.  Example  problems  are  also  shown  in  Reference  1, 
demonstrating  computational  procedures  for  applying  the  AD  method. 

In  following  the  flowchart,  if  the  Weibull  model  hasn’t  been  accepted  as  a  desired  model, 
then  a  test  for  the  normal  distribution  is  suggested, 

fn(x)  =  — -"in  /  exP  [  (t  A)2/2  <7 2  ]  dt,  (3) 

a  (2rt)  -« 

where  ft  is  the  mean  and  a  2  is  the  variance.  The  AD  tests  for  the  normal  model  is  similar 
to  the  test  for  the  Weibull.  The  procedure  used  to  identify  the  normal  model  is  also  in 
References  1  and  2.  It  should  be  noted  that  for  small  samples  reliable  identification  of  a 
model  to  represent  the  data  is  difficult  unless  some  prior  information  of  the  population  is 
known. 

If  the  Weibull  and  normal  models  are  rejected,  then  a  nonparametric  method  can  be  used 
to  compute  the  basis  value  (see  the  flowchart).  This  method  does  not  assume  any  parametric 
distribution,  as  described  above.  Therefore,  model  identification  is  not  required,  although 
application  of  the  method  can  often  result  in  overly  conservative  estimates  for  the  basis  value. 

The  conventional  nonparametric  method5 6 7  requires  a  minimum  of  29  values  in  order  to 
obtain  a  “B”-basis  value,  and  300  are  needed  for  the  “A”-basis  number.  This  report  presents 
a  method  for  obtaining  “A”  and  “B”  basis  values  for  any  sample  size.  The  method  is  a  modi¬ 
fication  of  the  Reference  7  procedure  involving  the  ordered  data  values  arranged  from  least 
to  largest  with  the  basis  value  defined  as: 

B  =  X(r)  (X(1)  /  X(r))K,  (4) 

where  X^r)  is  r,h  ordered  value  and  X(j)  is  the  first  ordered  number.  In  References  1  and  2, 
tables  for  r  and  K  values  are  tabulated  for  sample  sizes  n.  Note,  in  the  case  where  “A”  values 
are  required  for  small  sample  sizes,  it  is  suggested  that  nonparametric  methods  be  applied  unless 

5.  ANDERSON,  T.  W.,  and  DARLING,  D.  A  A  Test  of  GoodnessofFit.  J.  Am.  Statir  Aaaoc.,  v.  49,  1954,  p.  765-769. 

6.  CONOVER,  W.  J.  Practical  Nonparametric  Statistic*.  John  Wiley  and  Son*,  1980,  p.  111. 

7.  HANSON,  D.  L.,  and  KOOPMANS,  L.  H.  Tolerance  Limits  for  the  Class  of  Distributions  With  Increasing  Hazard  Rates.  Annals  of 
Mathematical  Statistics,  v.  35,  1964,  p.  1561-1570. 
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some  prior  information  of  the  model  is  known.  This  is  because  of  the  limited  information 
available  in  the  lower  tail  region  of  the  distribution,  which  can  result  in  erroneous  estimates 
of  the  reliability  numbers.  The  “A” -basis  value  is  often  used  in  design  where  a  single  load 
path  exists;  therefore,  it  is  essential  that  the  value  be  conservative. 

Weibull  Method  •  “B”-Basis  Value 

Returning  to  the  sequence  of  operations,  as  outlined  in  the  flowchart,  if  the  Weibull 
model  is  accepted,  then  determine  the  basis  value  from  the  following  relationship: 

B  =  a[ ln(l/PB)]1//f,  (5) 

/V  A 

where  /?  and  a  are  maximum  likelihood  estimates  of  the  shape  /?  and  scale  a  of  the  Weibull 
distribution.  That  is,  these  estimates  maximize  the  likelihood  function,  which  is  the  product 
of  probability  densities  for  the  Weibull  model  evaluated  at  each  of  the  n  data  values.  Tables 
for  PB,  as  a  function  of  the  sample  size  n  and  the  code  for  determining  a  and  /?,  are  given  in 
References  2  and  3. 

Normal  Method  -  “B”-Basis 

If  the  Weibull  model  was  rejected  and  the  normal  model  is  an  acceptable  representation 
of  the  data,  then  compute  the  basis  value  as: 


B  =  X  -  KbS,  (6) 

where  X  and  S  are  the  mean  and  SD,  and  KB  is  obtained  from  tables  in  References  1  and  2. 

PROCEDURES  FOR  MULTIPLE  BATCHES 

Anderson-Darling  Test 

If  there  are  more  than  one  batch  of  data  being  analyzed,  then  a  significance  test  is 
required  in  order  to  determine  if  the  batches  may  be  pooled  or  if  a  multi-batch  statistical  anal¬ 
ysis  is  to  be  applied  (see  the  flowchart).  Note,  the  outlier  test  is  to  be  applied  to  pooled 
data  prior  to  testing.  The  recommended  test  is  the  K-Sample  Anderson-Darling  Test1'8 9  which 
determines  if  batch-to-batch  variability  exists  among  the  K  batches.  This  test  is  simila'-  to  the 
AD  test  for  identifying  acceptable  statistical  models  for  representing  data.  In  the  K  sample 
case,  paired  comparisons  are  made  for  the  empirical  CDFs,  while  the  other  AD  methods  com¬ 
pare  a  parametric  CDF  with  an  empirical  CDF.  In  all  cases,  this  comparison  involves  the 
integration  of  the  squared  difference  of  the  CDFs  weighted  in  the  tail  region  of  the  distribu¬ 
tion.  The  K-Sample  AD  is  basically  a  two-sample  test  in  that  each  sample  (i'h  batch)  is  indi¬ 
vidually  compared  with  the  pooled  K-l  other  batches,  repeated  K  times  until  each  i,h  batch 
has  been  compared.  The  average  of  these  K  two-sample  tests  determines  the  K-Sample  AD 
test  statistic.  Tables  of  critical  values  and  a  detailed  description  of  the  method  and  its  appli¬ 
cation  are  shown  in  References  1,  2,  and  8. 

If  a  significant  difference  is  noted  among  the  K  batches,  then,  as  shown  in  the  flowchart, 
a  test  for  equality  of  variance  is  suggested  using  a  method  in  Reference  9.  Application  of 


8.  SCHOLZ,  F.  W.,  and  STEPHENS,  M.  A.  K- Simple  Anderson-Darting  Tests.  J.  Am.  Stalls.  Assoc,  v.  82,  1987,  p.  918. 

9.  LEHMANN,  E.  L.  Testing  Statistical  Hypothesis.  John  Wiley  and  Sons,  1959,  p.  274-275. 
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the  method,  tables,  and  the  necessary  relationships  for  computing  the  test  statistic  are  given 
in  References  1  and  2.  The  variance  test  is  suggested  only  as  a  diagnostic  tool.  Sample  test 
results  that  have  large  variances  relative  to  the  other  batches  may  identify  possible  problems 
in  testing  or  manufacturing  of  the  specimens.  Equality  of  variance  is  not  required  when 
applying  the  Modified  Lemon  method,  as  discussed  below,  in  the  multi-batch  case.  Although 
the  Modified  Lemon  method  is  based  on  the  assumptions  of  equality  of  variance  and  norma¬ 
lity,  simulation  results  have  shown  that  these  assumptions  are  not  necessary.  After  testing  for 
equality  variance,  it  is  suggested  that  the  basis  value  be  obtained  from  application  of  the 
Modified  Lemon  method  (see  Figure  1). 

The  Modified  Lemon  Method 

Composite  materials  typically  exhibit  considerable  variability  in  strength  from  batch  to 
batch.  Because  of  this  variability,  one  should  not  indiscriminately  pool  data  across  batches 
and  apply  single  batch  procedures.  The  K-Sample  Anderson-Darling  Test  was  introduced  into 
the  MIL- 17  Handbook  in  order  to  prevent  the  pooling  oi  data  in  situations  where  significant 
variability  exists  between  batches.  For  the  situation  where  the  K-Sample  Anderson-Darling 
Test  indicates  that  batches  should  remain  distinct,  a  special  basis  value  procedure  has  been 
provided.  This  method,  referred  to  as  the  “ANOVA”  or  “Modified  Lemon”  method,  will  be 
discussed  next.  A  detailed  description  for  applying  the  method  is  shown  in  References  1  and  2. 
For  a  discussion  of  the  underlying  theory,  see  Reference  10,  the  original  Lemon  paper,  and 
Reference  11,  the  Mee  and  Owen  paper  which  modifies  the  Lemon  method. 

The  Modified  Lemon  method  considers  each  strength  measurement  to  be  a  sum  of  three 
parts.  The  first  part  is  an  unknown  constant  mean.  If  one  were  to  produce  batches  end¬ 
lessly,  breaking  specimens  from  each  batch,  the  average  of  all  of  these  measurements  would 
approach  this  unknown  constant  in  the  limit  of  infinitely  many  batches.  Imagine,  however, 
that  one  were  to  test  many  specimens  from  a  single  batch.  The  average  strength  approaches 
a  constant  in  this  situation  as  well,  but  this  constant  will  not  be  the  same  as  for  the  case 
where  each  specimen  came  from  a  different  batch.  The  average  converges  to  an  overall  popu¬ 
lation  mean  (a  “grand  mean”)  in  the  first  case,  while  the  average  converges  to  the  population 
mean  for  a  particular  batch  in  the  second  case.  The  difference  between  the  overall  popula¬ 
tion  mean  and  the  population  mean  for  a  particular  batch  is  the  second  component  of  a 
strength  measurement.  This  difference  is  a  random  quantity,  it  will  vary  from  batch  to  batch 
in  an  unsystematic  way.  We  assume  that  this  random  variable  has  a  normal  distribution  with 
a  mean  of  zero  and  some  unknown  variance,  which  we  refer  to  as  the  “between  batch”  com¬ 
ponent  of  variance.  Finally,  in  order  to  arrive  at  the  value  of  a  particular  strength  measure¬ 
ment,  we  must  add  to  the  sum  of  the  con  tant  overall  mean  and  a  random  shift,  due  to  the 
present  batch,  a  third  component.  This  is  another  random  component  which  differs  for  each 
specimen  in  each  batch.  It  represents  variability  about  the  batch  mean.  It  also  is  assumed 
to  have  a  normal  distribution  with  a  mean  of  zero  and  an  unknown  variance,  which  is 
referred  to  as  the  “within  batch”  component  of  variance. 

The  “Modified  Lemon”  method  uses  the  data  from  several  batches  to  determine  a  mate¬ 
rial  basis  property  value  which  provides  95%  confidence  on  the  appropriate  percentile  of  a 
randomly  chosen  observation  from  a  randomly  chosen  future  batch.  This  basis  property 


10.  LEMON.  G.  H.  Factors  and  One-Sided  Tolerance  Bounds  fc  Balanced  One-Way  ANOVA  Random  Effects  Model.  J.  Am.  Slatis.  Assoc., 
v.  72,  1977,  p.  676-680. 

11.  MEE,  R.  W.,  and  OWEN,  D.  B.  Improved  Factors  for  One-Sided  Tolerance  Limits  for  Balanced  One-Way  ANOVA  Random  Effects  Model. 
J.  Am.  Statii.  Assoc.,  v.  78,  198i.  p.  901-905. 
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provides  protection  against  the  possibility  of  batch-to-batch  variability  resulting  in  future 
batches  which  have  lower  mean  strength  than  those  batches  for  which  data  are  available. 

To  see  what  this  means,  imagine  that  several  batches  have  been  tested  and  that  this 
statistical  procedure  has  been  applied  to  provide  a  “B”-basis  value.  Now,  imagine  that 
another  batch  was  obtained  and  a  specimen  tested  from  it.  After  this,  still  another  batch  was 
obtained  and  a  specimen  tested  from  it.  If  this  process  were  repeated  for  infinitely  many 
future  batches,  a  distribution  of  strength  measurements  corresponding  to  a  randomly  chosen 
measurement  from  a  random  batch  would  be  obtained.  There  would  be  a  95%  certainty  that 
the  basis  value  which  was  calculated  originally  is  less  than  the  tenth  percentile  of  this  hypo¬ 
thetical  population  of  future  measurements.  This  is  the  primary  reason  why  the  Modified 
Lemon  method  is  advocated  by  the  MIL-17  Handbook,  it  provides  protection  against  varia¬ 
bility  between  batches  which  will  be  made  in  the  future  through  the  use  of  data  which  is 
presently  available. 

An  illustrative  example  of  this  method  applied  to  nine  batches  of  material  is  shown 
below.  The  data  sets  did  not  pass  the  K-Sample  AD  Test  for  pooling.  Let  the  batches  be: 
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61.3 

66.5 

66.0 

61.9 

68.9 

75.8 

72.8 
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68.5 

64.7 

72.7 

68.0 

65.0 

75.2 

75.0 

71.0 

76.3 

62  5 

64.9 

67.1 

63.3 

70.9 

71.5 

66.3 

69.5 

76.6 

66.0 

65.2 

67.7 

74.6 

65.4 

69.6 

69.5 

69.5 

662 

66.6 

70.3 

65.7 

66.2 

66.5 

66.1 

71.9 

72.6 

72.4 

64.8 

68.2 

64.9 

74.6 

72.8 

69.5 

69.1 

109.6 

with  a  single  outlier. 

109.6  determined 

from 

MNR 

method. 

Let’s 

assume 

109.6  was  an  incor- 

rect  test  result  and  replaced  by  69.6,  e 

i  corrected  test  value. 

After  a  substantial  amount  of  computation1,2  involving  sums  of  squares,  within  batch  and 
between  batch  variances,  noncentral  t  distribution,  etc.,  the  “B”-basis  value  is: 

“B”  =  60.93. 


The  summary  statistics  are  shown  below. 
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It  should  be  noted,  the  value  of  60.93  is  lower  than  61.9  of  nonparametric  solution  from  the 
pooled  sample.  The  Modified  Lemon  method  can  be  overly  conservative  (low  basis  values)  in 
order  to  guarantee  90%  reliability  with  95%  confidence.  The  number  of  batches  and  the  vari¬ 
ability  between  and  within  the  batches  affect  the  computation  of  the  basis  value.  If  there  are 
few  batches  and  large  between  batch  variability  with  small  within  batch  variability,  then  this  sit¬ 
uation  could  result  in  very  low  basis  numbers,  depending  oq  the  amount  of  variability  and 
number  of  batches. 

In  Figure  4,  results  from  application  of  flowchart  procedures  are  shown  for  three  batches 
of  five  specimens  of  AS4/Epoxy  material  tested  in  compression.  In  this  case,  the  mean 
strength  values  show  a  small  amount  of  variability,  while  there  is  a  relatively  large  spread 
within  each  data  set.  “B”-basis  results  from  the  flowchart  application  are  for  the  following: 
ANOVA  (Modified  Lemon),  Weibull,  normal,  lognormal,  and  nonparametric  methods.  A  list 
of  assumptions  that  were  violated  are  not  included  in  the  flowchart  results.  The  results  show 
a  small  difference  in  basis  values,  except  for  the  nonparametric  solution  which  has  the  low 
value  of  167.1.  The  Weibull  method  was  suggested  since  it  passed  the  K-Sample  AD  Test 
and  the  AD  goodness-of-fit  test.  The  relatively  large  within  batch  variances  and  small  differ¬ 
ences  in  mean  values  made  it  possible  to  pool  the  batches. 

Batch  1:  Mean  strength  •  221  Ksi 

Batch  2:  Mean  strength  *  222  Ksi 

Batch  3;  Mean  strength  ■  220  Ksi 
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200  20S  210  2  IS  220  225  230 

235 

METHOD 

BASIS  .VALUE 

ANOVA 

202.6 

Ksi 

Weibull 

196.5 

Ksi 

Normal 

199.1 

Ksi 

Lognomal 

199.6 

Ksi 

Nonparametric 

167.1 

Ksi 

The  Weibull  result  is 

recommended 

by 

(3  batchai  of  S,  AS4/Epo«y  compraitlon) 

Figure  4.  Example  of  basis  value  calculation:  Negligible 
batclvto- batch  variability. 

Figure  5  shows  another  result  of  computing  the  “B”-basis  values  using  the  ANOVA, 
Weibull,  and  normal  methods  applied  to  another  three  selected  batches  from  the  same  popula¬ 
tion,  as  shown  in  Figure  4.  The  ANOVA  result  of  15.7  ksi  is  substantially  lower  than  those 
from  the  other  two  methods.  Unfortunately,  this  is  a  result  of  a  large  difference  in  mean  val¬ 
ues  preventing  pooling  of  the  batches  resulting  in  the  required  ANOVA  application.  The 
large  difference  in  mean  values,  in  addition  to  relatively  small  within  batch  variability,  resulted 
in  this  extremely  low  basis  value.  A  “B”  value  of  6.5  was  obtained  from  the  simple  normal 
analysis  using  the  three  mean  values.  The  result  shows  that,  for  this  example,  the  ANOVA 
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method  primarily  depends  on  the  batch  means.  The  above  results  would  suggest  obtaining 
more  batches  or  investigating  testing  and  processing  procedures. 

Batch  1:  Mean  strength  •  181  Ksi 

Batch  2:  Mean  strength  •  236  Ksi 

Batch  3:  Mean  strength  ■  241  Ksi 
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METHOD 
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ANOVA 

15.7 

K3i 

Weibull 

161.9 
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The  ANOVA  result  is  recommended  by 
the  Flowchart.  Normal  analysis  using 
only  the  three  mean  values  gives  a  B- 
basis  value  ol  6.5.  Either  reiect  the 
material  as  loo  variable  or  obtain  more 
batches. 


(3  batch**  of  S,  AS4/Epoxy  compression) 

Figure  5.  Example  of  basis  value  calculation:  Substantial 
batch-to-batch  variability. 


In  Figure  6,  results  are  shown  for  the  case  of  randomly  selecting  another  batch  from  the 
same  population  described  in  Figure  5.  In  this  case,  the  ANOVA  result  shows  a  value  of 
105.4  ksi,  which  is  substantially  larger  than  the  15.7  ksi  recorded  for  the  three  batches.  The 
importance  in  having  a  larger  number  of  batches  is  shown  from  these  results  in  Figures  5  and  6. 
Also,  with  more  data  available,  the  pooled  results  for  the  Weibull  and  normal  model  also 
resulted  in  less  conservative  values. 

Figure  7  presents  results  showing  where  a  substantial  amount  of  within  batch  data  is  not 
necessary.  In  Case  1,  the  ANOVA  results  for  three  batches  of  100  data  values  each,  resulted 
in  154.9  ksi,  while  for  Case  2,  three  batches  of  ten  each,  a  “B”-basis  value  of  152  ksi  was 
obtained.  This  result  emphasizes  the  importance  of  being  able  to  obtain  more  batches  rather 
than  increasing  the  batch  size.  However,  the  ANOVA  results  in  Figure  4  show  three  batches 
can  provide  reasonable  results  similar  to  pooled  results  if  small  differences  in  mean  values  rela¬ 
tive  to  batch  variances  exist.  Note  that  for  very  large  batch  sizes,  the  K-Sample  AD  Test 
can  reject  pooling  of  data  even  though  there  is  a  small  difference  in  mean  values.  This  rejec¬ 
tion  is  statistically  correct,  but  the  user  of  the  flowchart  may  consider  the  difference  in  the 
batch  means  not  of  engineering  importance.  In  this  case,  the  user  can  make  the  decision  of 
pooling  or  not  pooling,  since  there  will  be  a  small  difference  in  basis  values  from  pooled  or 
unpooled  results.  If  there  are  large  batch  differences  and  the  ANOVA  method  is  suggested 
from  the  flowchart,  then  adding  more  batches  can  reduce  the  conservatism.  The  ANOVA 
method  is  a  random  effects  model  which  determines  a  basis  value  representing  all  future 
values  obtained  from  the  same  material  system  and  type  of  test.  In  order  to  provide  this 
guarantee  in  the  presence  of  large  batch-to-batch  variability,  there  is  the  potential  for  it  to 
be  overly  conservative,  which  was  shown  in  Figure  5. 
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Batch  1:  Mean  strength  *  181  Ksi 

Batch  2:  Mean  strength  •  236  Ksi 

Batch  3:  Mean  strength  *  24  1  Ksi 

Batch  4:  Mean  strength  *  217  Ksi 
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The  ANOVA  result  is  recommended  by 
the  Flowchart.  A  single  additional  batch 
increased  the  basis  value  from  15.7  Ksi 

to  105.4  Ksi. 


(4  batch**  ol  5,  AS4/Epo*y  comprataion) 


Figure  6.  Example  of  basis  vaiua  calculation:  The  effect  of  an 
additional  batch. 


Case  1: 

T300/Epoxy  Unidirectional  Tension 
3  batches  ol  100  specimens  each 

Method  Basis  Value 

ANOVA  154.9 

Weibull  171.7 

Normal  175.7 

Qggg  2: 

One  random  dataset  ol  10  Irom  each 
ol  the  above  three  batches. 

Method  Basis  Value 

ANOVA  152.0 

Weibull  165.7 

Normal  172.5 

The  ANOVA  method  is  recommended  by  the 
Flowchart.  Note  that  there  is  little 
difference  between  basis  values 
lor  batch  sizes  o(  10  and  basis  values 
lor  batch  sizes  ol  100. 


Figure  7.  The  effect  of  increased  batch  size:  Substantial 
between  batch  variability. 
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Reliability  at  Basis  Stress  Value 


Figure  8  conceptually  describes  the  statistical  reliability  of  a  simple  structure  in  tension  as 
it  relates  to  the  “B”-basis  applied  stress  value.  In  the  example  shown  in  the  figure,  ten  per¬ 
cent  of  all  the  specimens  (structures)  will  fail  when  subjected  to  load  S.  This  statement 
should  be  incorrect  at  most  one  time  in  twenty  (95%  confidence).  S  is  the  “B”-basis  value 
obtained  from  strength  (failure  load)  measurements  from  specimens  of  the  same  material  and 
geometry.  This  statistical  guarantee  that  at  most  10%  of  the  specimens  will  fail,  can  provide 
the  engineer  with  a  quantitative  number  for  selecting  and  applying  materia!  in  composite  mate¬ 
rial  structures.  This  is  unlike  the  conventional  deterministic  property  value  approach  which  is 
an  ad  hoc  procedure  that  reduces  the  mean  strength  measurements  in  order  to  obtain  some 
design  value  which  can  result  in  a  potentially  over  or  under  design  situation.  In  applying  the 
statistical  basis  value,  it  is  assumed  the  material,  geometry,  and  loading  conditions  in  the  struc¬ 
tural  design  situation  is  similar  to  those  obtained  from  the  strength  measurements.  This  is 
also  true  for  deterministic  property  value  applications.  In  the  following  sections  the  inadequa¬ 
cies  of  the  deterministic  approach  are  discussed  in  more  detail. 


The  (clUblllty  o<  c  tail  tpcclmcn  at  the  3-ttilt  ilim 
■hould  be  high.  For  a  cUtlaticaliy  baaed  B-  baala  valua 
calculated  from  a  procedure  appropriate  to  the  data,  thie 
reliability  la  guaranteed  to  bo  at  laaat  30% 

(I, a.,  00%  with  00%  conlldonce). 


S  ■  B -basis  stress 


S 


N  apaclmana  ol  which  F  tall  al  or  below  alraaa  S. 
Eellmated  reliability  at  0-baale  alraaa  (N-F)/N 

Figure  8.  Reliability  at  baaie  streaa:  Statistical 
versus  deterministic. 


Reliability  Values  -  Statistical  Versus  Deterministic 

In  Figure  9,  the  results  of  a  simulation  process  involving  the  random  selection  of  ten 
values  from  a  population  of  191  strength  measurements  repeated  2,500  times,  are  graphically 
displayed.  For  each  simulation  a  design  number  or  material  property  value  is  obtained  from 
each  of  thejthree  procedures  X/2,  (2/3)X,  and  the  MIL- 17  flowchart.  The  mean  value  of  the 
data  set  is  X.  The  reliability  values,  as  shown  in  the  figure,  are  obtained  by  evaluating  the 
population  probability  distribution  fit  to  the  191  values  at  the  design  numbers. 

In  the  case  where  the  mean  is  reduced  by  a  factor  of  1/2,  the  strength  values  are  very 
low  (90  ksi)  and  the  reliability  is  extremely  high  (1.0).  The  engineer  may  not  be  able  to 
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afford  such  a  high  reliability  value  of  1.0  (to  twenty  significant  digits)  at  the  expense  of 
having  design  values  as  low  as  90  ksi  when  mean  strength  is  180  ksi.  The  factor  of  2/3 
increases  the  design  value  but  reduces  the  reliability  to  approximately  0.999.  The  flowchart 
“B”-basis  calculation  provides  higher  strength  values  with  acceptable  reliability  numbers.  The 
other  two  procedures  show  an  element  of  uncertainty  by  depending  on  the  chosen  factor.  If 
the  engineer  used  the  factor  of  1/2,  this  would  result  in  an  extreme  over-design  situation 
requiring  either  rejection  of  the  material  or  the  design.  Alternatively,  if  the  engineer  used 
the  mean  strength  as  the  design  number,  the  reliability  would  be  reduced  to  0.5,  although 
strength  values  would  be  much  higher.  The  flowchart  procedure  removes  the  uncertainty  by 
providing  a  guaranteed  minimum  reliability  of  0.90,  without  unnecessarily  reducing  the  basis 
value.  The  minimum  reliability  can  be  increased  to  0.99,  if  necessary,  by  using  “A” -basis 
computations,  as  outlined  in  the  MIL-17  Handbook. 


Population:  191  drangth  valuoa 

Dataaot:  10  apaclmana  clioaan  2S00  tlmaa  randomly 
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0  Reliability  Valuoa 

(T300/Epoay  Unidirectional)  TENSION 

Figure  9.  Reliability/strength  comparison:  A  case  study  • 
statistical  versus  deterministic. 


Effects  of  Variance  on  Reliability  Estimates 

In  Figure  10,  the  effects  of  variance  differences,  as  they  relate  to  reliability  estimates,  are 
shown  from  a  simulation  process.  This  involved  randomly  selecting  ten  values  from  each  of 
two  separate  normal  distributions  with  the  same  mean  of  100  and  different  SDs  of  5  and  25 
repeated  2,500  times.  The  reliability  values  are  obtained  in  a  similar  manner,  as  described  in 
the  previous  section,  except  the  probability  values  were  obtained  from  the  normal  distribution. 
In  the  case  where  the  SD^  is  5,  there  is  very  little  dispersion  in  the  reliability  values.  Again, 
the  design  number  from  X/2  is  substantially  lower  than  the  basis  value  using  the  flowchart  pro¬ 
cess,  although  the  reliability  is  very  high  for  this  number.  Note  that  when  SD  is  25,  there  is 
a  substantial  increase  in  the  dispersion  of  the  reliability  values,  particularly  for  the  basis  value 
using  the  flowchart  method.  The  flowchart  results  show  similar  reliability  estimates  for  both 
SDs  of  5  and  25,  although  for  the  X/2  the  reliability  has  been_ reduced  substantially  from 
twelve  nines  to  0.96.  This  is  the  result  of  the  deterministic  (X/2)  approach  being  indepen¬ 
dent  of  variance.  This  is  not  an  issue  if  50%  reliability  is  required,  but  for  90%  reliability, 
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variability  is  important.  Dividing  the  mean  by  two  can  be  nonconservative  for  a  situation 
when  the  distribution  has  a  large  spread  (long  tail).  In  order  to  made  an  adjustment  for  this 
situation,  the  flowchart  method  (basis  value)  is  suggested  (see  the  results  in  the  figure  where 
the  basis  value  adjusts  to  a  lower  level  but  maintains  the  same  range  for  the  reliability  esti¬ 
mates).  The  basis  value  will  guarantee  a  reliability  by  adjusting  the  design  value,  while  the 
safety  factor  approach  cannot  guarantee  reliability.  This  result  suggests  using  the  basis 
method  if  it  is  important  to  maintain  a  certain  level  of  reliability.  The  overall  issue  is  that 
the  flowchart  methods  will  provide  property  values  with  specified  reliability  with  95%  confi¬ 
dence,  while  the  deterministic  approach  is  an  ad  hoc  approach  with  no  control  of  the  resul¬ 
ting  reliability  estimates. 
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Figure  10.  Reliability/strength  comparison:  A  case  study- 
statistical  versus  deterministic. 


CONCLUSIONS 

This  report  is  an  exposition  of  the  statistical  procedures  described  in  the  MIL- 17  Hand¬ 
book  for  obtaining  material  property  values.  Its  primary  goal  was  to  introduce  the  MIL-17 
statistics  chapter  to  the  users  so  that  they  may  use  it  more  effectively.  The  methods  and  the 
sequence  of  operations,  suggested  by  the  statistics  chapter  flowchart,  were  analyzed  with 
respect  to  their  effectiveness,  purpose,  and  limitations.  By  following  the  flowchart  procedures, 
guidance  is  provided  to  the  user  so  that  reasonably  accurate  property  values  may  be  obtained 
without  relying  on  ad  hoc  schemes  which  could  potentially  result  in  either  excessively  low  or 
high  values. 

Each  method  and  its  order  of  application  were  discussed  with  respect  to  their  specific  pur¬ 
pose,  such  as  model  identification,  batch-to-batch  variability  recognition,  outlier  detection,  and 
the  basis  value  computation.  There  are  situations  where  low  basis  values  will  result,  not 
because  of  limitations  in  the  statistical  procedures,  but  are  usually  the  result  of  very  large  or 
small  data  sets,  large  batch-to-batch  variations,  or  model  recognition. 

The  comparison  between  the  statistical  reliability  and  the  deterministic  approach  showed  a 
preference  for  statistics  since  it  was  able  to  guarantee  a  specified  reliability  in  contrast  to  a 
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deterministic  method,  which  is  primarily  an  ad  hoc  process  resulting  in  considerable  uncer¬ 
tainty  as  to  the  corresponding  reliability  estimates.  Finally,  the  authors  have  attempted  to 
provide  a  satisfactory  definition  of  a  statistically-based  material  property  value  by  introducing 
the  tolerance  limit  concept  and  its  importance.  A  number  of  illustrations  were  presented 
showing  the  advantage  of  the  tolerance  limit  over  the  deterministic  approach. 
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